Here's how it will go down. They will pester you until you agree to a visit. You could try and tell them that you'd rather not meet, doing fine thanks but that will raise a small flag. These boys have sniffed out, hunted out and DFed their fair share of apostates. They are experienced and know all of the signs. You could probably fool your local body, but not these cats. At all costs, defer your visit til they're gone.
Other side of the coin: you meet, have a nice family shep visit. They will ask you about your family study schedule. They will ask you how often you pray. Either your answer will be never, or your wife and kid get to sit and watch you LIE to these mens faces. There's no way you can get through a shep visit with them without either giving them too much info, or lying to them. Both are bad outcomes.
In my humble opinion, your best option for the time being is to just take the risk and tell them you're not available until their visit is over. Then when your local elders ask you when would be a good time to pop by, tell them you'd rather not. That should buy you 6 months or so.